Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix problematic ERS cases #8422

Merged
merged 5 commits into from
Jul 7, 2021
Merged

Fix problematic ERS cases #8422

merged 5 commits into from
Jul 7, 2021

Conversation

aquarapid
Copy link
Contributor

Already approved as #8319 ; redone to clean up the commit list which was polluted with hundreds of merge commits.

Description

Checklist

  • Tests were added or are not required
  • Documentation was added or is not required

deepthi
deepthi approved these changes Jul 6, 2021
View reviewed changes
Copy link
Member

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming all the logging is intentional. I don't have a problem with it, it should help with analysis when things go wrong.
LGTM overall.
How do you feel about squashing the commits together?

@deepthi
Copy link
Member

deepthi commented Jul 6, 2021

I forgot this while reviewing..
Ideally we should fix this on the earliest branch where we think it should be fixed. I think in this case that is release-8.0
@systay does that sound right? I think the PR needs to be re-done against release-8.0, correct?

aquarapid added 5 commits July 7, 2021 07:27
@aquarapid @systay
Make it clearer which GTIDSet we are using.
5832cd4 
Signed-off-by: Jacques Grove <[email protected]>
@aquarapid @systay
The difference between two identical GTIDSet values should be empty, not
3f6ff25 
just a GTID UUID/SID without an offset/interval.

Signed-off-by: Jacques Grove <[email protected]>
@aquarapid @systay
We use a sentinel file to avoid replication being restarted
5a4c504 
automatically in some cases (e.g. ERS);  however, we were not resetting
this sentinel flag in at least two cases:  1) When we have re-parented
successfully (setMasterLocked) and 2) When we have successfully
promoted a replica (PromoteReplica)

Signed-off-by: Jacques Grove <[email protected]>
@aquarapid @systay
Add additional logging for most of the tablet replication RPC endpoints,
8b0f5e3 
will make it easier to debug from log output.

Signed-off-by: Jacques Grove <[email protected]>
@aquarapid @systay
Add test for a series of trivial ERS failovers to catch problems like
5d06df2 
Signed-off-by: Jacques Grove <[email protected]>
@aquarapid aquarapid requested a review from rohit-nayak-ps as a code owner July 7, 2021 05:33
@systay systay changed the base branch from main to release-8.0 July 7, 2021 05:33
@systay systay requested a review from sougou as a code owner July 7, 2021 05:33
@deepthi deepthi merged commit ea9c632 into vitessio:release-8.0 Jul 7, 2021
@deepthi deepthi deleted the jg_gtidset branch July 7, 2021 17:02
This was referenced Jul 14, 2021
Closed
Closed
@ajm188 ajm188 mentioned this pull request Jul 18, 2021
Merged
deepthi added a commit that referenced this pull request Aug 31, 2021
@deepthi
Merge pull request #8744 from planetscale/jg_ers_11
10630c5 
Port #8422 to 11.0 branch
deepthi added a commit that referenced this pull request Aug 31, 2021
@deepthi
Merge pull request #8745 from planetscale/jg_ers_main
a3b0908 
Port #8422 to main branch
@deepthi deepthi mentioned this pull request Sep 9, 2021
Merged
@GuptaManan100 GuptaManan100 mentioned this pull request Nov 21, 2021
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deepthi deepthi deepthi approved these changes

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal

@shlomi-noach shlomi-noach Awaiting requested review from shlomi-noach

@systay systay Awaiting requested review from systay

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps

@sougou sougou Awaiting requested review from sougou

Assignees

@systay systay

Labels
Component: Cluster management Type: Bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

ERS does not clear do_not_replicate flag on disk in some cases ERS seems to break in errant GTID check
3 participants
@aquarapid @deepthi @systay